{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sFaB2n259LyH"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7z5MNErn9hLk"
   },
   "source": [
    "# Tensorflow2.0教程-自定义层\n",
    "\n",
    "tensorflow2.0建议使用tf.keras作为构建神经网络的高级API。 也就是说，大多数TensorFlow API都可用于eager执行模式。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3ef8wxGvJzY3"
   },
   "outputs": [],
   "source": [
    "from __future__ import absolute_import, division, print_function, unicode_literals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 107
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 45625,
     "status": "ok",
     "timestamp": 1559141517811,
     "user": {
      "displayName": "Will Chen",
      "photoUrl": "",
      "userId": "01179718990779759737"
     },
     "user_tz": -480
    },
    "id": "-jisbMBWKMsO",
    "outputId": "98edad8c-80fc-42c2-d42d-bdcae392d98a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[K     |████████████████████████████████| 79.9MB 1.4MB/s \n",
      "\u001b[K     |████████████████████████████████| 61kB 20.4MB/s \n",
      "\u001b[K     |████████████████████████████████| 3.0MB 38.8MB/s \n",
      "\u001b[K     |████████████████████████████████| 419kB 46.6MB/s \n",
      "\u001b[?25h2.0.0-alpha0\n"
     ]
    }
   ],
   "source": [
    "!pip install -q tensorflow==2.0.0-alpha0\n",
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "T85cnaZtKdPF"
   },
   "source": [
    "## 一、网络层layer的常见操作\n",
    "通常机器学习模型可以表示为简单网络层的堆叠与组合，而tensorflow就提供了常见的网络层，为我们编写神经网络程序提供了便利。\n",
    "TensorFlow2推荐使用tf.keras来构建网络层，tf.keras来自原生keras，用其来构建网络具有更好的可读性和易用性。\n",
    "\n",
    "如，我们要构造一个简单的全连接网络，只需要指定网络的神经元个数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-w9d8-2wKYHy"
   },
   "outputs": [],
   "source": [
    "layer = tf.keras.layers.Dense(100)\n",
    "# 也可以添加输入维度限制\n",
    "layer = tf.keras.layers.Dense(100, input_shape=(None, 20))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "szlH18xYNwgM"
   },
   "source": [
    "可以在[文档](https://www.tensorflow.org/api_docs/python/tf/keras/layers)中查看预先存在的图层的完整列表。 它包括Dense，Conv2D，LSTM，BatchNormalization，Dropout等等。\n",
    "\n",
    "每个层都可以当作一个函数，然后以输入的数据作为函数的输入"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "S0QZ2EN9NfGy"
   },
   "outputs": [],
   "source": [
    "layer(tf.ones([6, 6]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MNSBfM3EPFFa"
   },
   "source": [
    "同时我们也可以得到网络的变量、权重矩阵、偏置等 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 2184
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1695,
     "status": "ok",
     "timestamp": 1558271994963,
     "user": {
      "displayName": "Will Chen",
      "photoUrl": "",
      "userId": "01179718990779759737"
     },
     "user_tz": -480
    },
    "id": "m67fg_IeO-Rj",
    "outputId": "afc51859-8048-4a92-babb-c8831dfc9e09"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<tf.Variable 'dense_1/kernel:0' shape=(6, 100) dtype=float32, numpy=\n",
      "array([[-0.18237606,  0.16415142,  0.20687856,  0.23396944,  0.09779547,\n",
      "        -0.14794639, -0.10231382, -0.22263053, -0.0950674 ,  0.18697281,\n",
      "         0.20488529, -0.04037735, -0.19727436,  0.0979359 , -0.1759503 ,\n",
      "         0.22504129,  0.21929003, -0.1273948 , -0.13652515,  0.02981101,\n",
      "         0.14656503,  0.20608391,  0.14076535, -0.02625689, -0.00161622,\n",
      "        -0.01449171,  0.23303385,  0.14593105,  0.11570902, -0.03970808,\n",
      "        -0.05525994, -0.20392904, -0.10306785,  0.21736331,  0.10087742,\n",
      "        -0.14146385,  0.03447478,  0.01457174, -0.06794603,  0.1030371 ,\n",
      "        -0.15175559,  0.22587933,  0.0804611 ,  0.21479838, -0.11029668,\n",
      "         0.22146653, -0.07499251,  0.1368954 , -0.13015983, -0.12019924,\n",
      "         0.21677957, -0.09586674, -0.05949883,  0.22539525,  0.2289827 ,\n",
      "        -0.02051648,  0.01296295,  0.16009761,  0.10034381,  0.12798755,\n",
      "        -0.10539538,  0.11883061,  0.07966466, -0.22101976,  0.12746729,\n",
      "        -0.1093536 , -0.16521278,  0.20071043,  0.16937451,  0.01447372,\n",
      "         0.16793476, -0.13962969,  0.1615852 , -0.10127702, -0.21089599,\n",
      "        -0.03635107, -0.2252161 , -0.02891247,  0.04012387,  0.1437303 ,\n",
      "        -0.14835042,  0.04761215,  0.00950299, -0.23300804,  0.09713729,\n",
      "         0.15262072, -0.00947247,  0.07256009, -0.15564013, -0.23770826,\n",
      "         0.20197298,  0.17501004,  0.16743289, -0.05297002,  0.06925295,\n",
      "         0.13787319, -0.00939476,  0.21161182, -0.14816652,  0.09728603],\n",
      "       [-0.23597604,  0.09226345, -0.21754897,  0.030596  , -0.02821516,\n",
      "        -0.11382222,  0.04664303,  0.03997506,  0.11674343,  0.17904802,\n",
      "         0.09352373, -0.06271012, -0.0995118 , -0.0839863 ,  0.19747855,\n",
      "         0.20034815, -0.00912318, -0.07400802,  0.1354406 , -0.10645141,\n",
      "         0.01427723, -0.23239797, -0.12351944,  0.20848687,  0.23319058,\n",
      "        -0.23722333, -0.05799152,  0.16170807,  0.03911252, -0.15837833,\n",
      "         0.2375672 , -0.0743922 ,  0.14655517,  0.08806615,  0.0797369 ,\n",
      "         0.1290553 , -0.08087416,  0.15930201, -0.00874732,  0.05047153,\n",
      "        -0.1929135 , -0.13928278,  0.17150857, -0.12328131,  0.18083079,\n",
      "        -0.00578046, -0.07798126,  0.0491005 , -0.18679146,  0.15352319,\n",
      "         0.08727174, -0.05599469,  0.23526688, -0.16455042, -0.1487674 ,\n",
      "        -0.14412951,  0.13929401, -0.09632155,  0.00167181,  0.1395758 ,\n",
      "        -0.04055098,  0.18666495, -0.02553804,  0.02356009,  0.08105092,\n",
      "         0.16715266,  0.08995526, -0.22296879,  0.06449543,  0.12188949,\n",
      "         0.23288779,  0.02098565, -0.07010749, -0.11215921,  0.23389243,\n",
      "         0.06119139, -0.0860803 , -0.12721145, -0.21600002, -0.20084816,\n",
      "        -0.16984203, -0.09152178, -0.1525074 , -0.09145156,  0.13034134,\n",
      "        -0.02183297,  0.04665013,  0.08182736, -0.18721978,  0.19582637,\n",
      "         0.02724861,  0.0457405 , -0.04569745,  0.04170756,  0.00739618,\n",
      "         0.06376933, -0.10515776,  0.15959997, -0.0694766 ,  0.02755009],\n",
      "       [-0.06319267,  0.13182057,  0.05506279, -0.20477852,  0.18970369,\n",
      "        -0.03943053,  0.11903851, -0.07155791,  0.04669012,  0.00812705,\n",
      "         0.05584623, -0.08371873, -0.15374976,  0.11677746, -0.12471052,\n",
      "        -0.16164276,  0.07642011,  0.20128562, -0.0017367 ,  0.1942391 ,\n",
      "        -0.00764962,  0.03465028, -0.18169758, -0.00213711,  0.05457829,\n",
      "        -0.07720377,  0.07186998, -0.16455126, -0.10122336,  0.17055278,\n",
      "         0.15423344,  0.23643543,  0.21533255, -0.08121212, -0.04754753,\n",
      "         0.08294679, -0.21654445, -0.01131117, -0.15386513, -0.06523256,\n",
      "        -0.19633041,  0.0248922 , -0.21637684,  0.00159171, -0.07713228,\n",
      "         0.22782163, -0.00349559, -0.16974016, -0.12448873,  0.17258836,\n",
      "        -0.01981039,  0.05522637,  0.12275855, -0.04751714,  0.05332409,\n",
      "        -0.07889435, -0.18033162,  0.06372721,  0.08295502, -0.19362703,\n",
      "        -0.0565086 ,  0.0755374 , -0.04072852,  0.07242356, -0.13613802,\n",
      "         0.20004983,  0.21880187,  0.14566241, -0.12664501,  0.21120678,\n",
      "         0.07900251,  0.00621621, -0.12596728,  0.11392109,  0.18359078,\n",
      "         0.07022355,  0.10555948, -0.22412848, -0.12310904,  0.1573142 ,\n",
      "        -0.11818868,  0.16200732,  0.15375106, -0.06835161, -0.2047078 ,\n",
      "         0.04541059,  0.1585447 ,  0.05231826,  0.22327141, -0.06841837,\n",
      "        -0.18981642,  0.08554949, -0.07791071,  0.12778889, -0.05718628,\n",
      "        -0.15301189,  0.17464243, -0.02707547,  0.23768233, -0.17579341],\n",
      "       [-0.09624396,  0.06217782, -0.02595268,  0.18477325, -0.10499094,\n",
      "         0.02485277, -0.13052183,  0.14919181,  0.13326867, -0.02644244,\n",
      "        -0.05067481,  0.218336  ,  0.07208259,  0.11023434,  0.09559919,\n",
      "         0.1890675 ,  0.16497074,  0.10391684, -0.10114242, -0.21803345,\n",
      "         0.20411105,  0.13692127, -0.15716076,  0.15078364,  0.04168098,\n",
      "        -0.21071486, -0.13353205, -0.16932839, -0.1560562 , -0.10680643,\n",
      "        -0.16614452,  0.10001676, -0.21421039, -0.06145182,  0.19260494,\n",
      "         0.17225821, -0.08490936, -0.05574624, -0.08104928, -0.0988984 ,\n",
      "         0.02178408, -0.16698101, -0.21433915,  0.0558214 , -0.00975212,\n",
      "         0.10072403, -0.02631079,  0.10000359,  0.15034248,  0.13407217,\n",
      "        -0.00151129, -0.0836809 ,  0.18076299,  0.15085675, -0.23618354,\n",
      "         0.05001326, -0.18851772, -0.11100499, -0.15566462, -0.20365193,\n",
      "        -0.2248187 , -0.07087977, -0.1306392 ,  0.0809411 , -0.00025243,\n",
      "         0.22963573,  0.01235358, -0.17157783, -0.2256927 , -0.17929551,\n",
      "         0.21892004, -0.23259266, -0.11126326, -0.08467883, -0.10183315,\n",
      "         0.2306784 , -0.09867099, -0.10394485, -0.10634935,  0.03752603,\n",
      "         0.17235063,  0.06114869,  0.022352  ,  0.15641572,  0.15094145,\n",
      "        -0.19431974,  0.15790848, -0.11217503,  0.19442542,  0.22548787,\n",
      "        -0.00768501,  0.158823  ,  0.14735304,  0.0210426 , -0.21737437,\n",
      "         0.08778961, -0.04887612,  0.0595323 ,  0.14631622,  0.02791832],\n",
      "       [-0.2108851 , -0.20682064, -0.20408219, -0.12765911, -0.06276481,\n",
      "        -0.13709348,  0.21450092,  0.01130743, -0.08537285,  0.14092736,\n",
      "         0.05730419, -0.02708216, -0.15316312,  0.00443964,  0.02637617,\n",
      "         0.10603566,  0.04577096, -0.11787698,  0.1437973 ,  0.23720677,\n",
      "        -0.1250069 , -0.03472549,  0.14871027, -0.05276214, -0.01313959,\n",
      "        -0.01017034,  0.09595035, -0.04737265,  0.10132308, -0.1576525 ,\n",
      "         0.09889011,  0.0852475 , -0.07178378, -0.191459  , -0.15800317,\n",
      "         0.21022727, -0.22400676, -0.2242892 ,  0.20423825,  0.2276028 ,\n",
      "         0.10706128, -0.15622596, -0.13000225, -0.03294952,  0.15762375,\n",
      "         0.09382527, -0.14523874,  0.08037744, -0.00737762,  0.16303493,\n",
      "         0.21322279, -0.20615923,  0.22627167, -0.01579981,  0.01377739,\n",
      "        -0.05311473,  0.0315261 , -0.05921595,  0.17797594,  0.21337356,\n",
      "        -0.00148429, -0.2224096 ,  0.17472763, -0.02621673,  0.12086569,\n",
      "         0.2221802 , -0.21260785,  0.19889398,  0.16766582,  0.11372416,\n",
      "         0.22781326, -0.17956433,  0.15436538,  0.20817696,  0.20706873,\n",
      "         0.17290573,  0.17291696,  0.0859191 , -0.17851931, -0.01899339,\n",
      "         0.06818415, -0.02687207, -0.09205528,  0.23415436,  0.09353746,\n",
      "        -0.09337692, -0.22046927,  0.20821108, -0.07228482,  0.04738481,\n",
      "        -0.13621926, -0.19762638, -0.13899282, -0.08098926,  0.130073  ,\n",
      "        -0.10348159, -0.07493602, -0.1722112 , -0.23290877,  0.18784209],\n",
      "       [ 0.13477843,  0.11936818, -0.21257897,  0.21244659, -0.18786472,\n",
      "        -0.06494723, -0.07063387, -0.07994832, -0.11256738, -0.22335076,\n",
      "        -0.02153319, -0.20943552, -0.21425952, -0.12278055, -0.00619341,\n",
      "        -0.09176037, -0.1766775 , -0.21622379, -0.04250833,  0.23764552,\n",
      "         0.21168886,  0.09459655, -0.07919639, -0.21559525,  0.20465617,\n",
      "        -0.20613717,  0.13103445,  0.21384992,  0.04693423,  0.20122723,\n",
      "         0.12190209,  0.22194327, -0.05410977, -0.1792583 , -0.03342254,\n",
      "         0.09272121,  0.06039228,  0.09666802, -0.22759588, -0.14688678,\n",
      "         0.12520896,  0.15474696, -0.23104139,  0.18017791, -0.02388267,\n",
      "        -0.01371126,  0.2352383 , -0.10501392,  0.01626216, -0.14222105,\n",
      "         0.13740788,  0.18499441,  0.03618436, -0.01862051, -0.1401035 ,\n",
      "        -0.01304157, -0.04905747, -0.07051091,  0.10759439, -0.08964662,\n",
      "        -0.01344521, -0.17841959, -0.17568308, -0.12892699,  0.11976974,\n",
      "         0.02280475,  0.16669382,  0.21027894,  0.21428709, -0.04820213,\n",
      "        -0.22136293, -0.13934767,  0.142024  , -0.07064074,  0.1470062 ,\n",
      "         0.00042979, -0.2371952 , -0.06649312,  0.10123204, -0.20473264,\n",
      "        -0.09161748,  0.20804678, -0.22195774, -0.09219673,  0.02311908,\n",
      "         0.13456099,  0.14470674, -0.05369592,  0.02126037,  0.0682667 ,\n",
      "         0.08384518,  0.17998771, -0.1927835 , -0.11473013, -0.01386146,\n",
      "        -0.10450925, -0.12111329, -0.2259491 ,  0.12304659, -0.04047236]],\n",
      "      dtype=float32)>, <tf.Variable 'dense_1/bias:0' shape=(100,) dtype=float32, numpy=\n",
      "array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "      dtype=float32)>]\n"
     ]
    }
   ],
   "source": [
    "print(layer.variables) # 包含了权重和偏置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 2184
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1120,
     "status": "ok",
     "timestamp": 1558272093654,
     "user": {
      "displayName": "Will Chen",
      "photoUrl": "",
      "userId": "01179718990779759737"
     },
     "user_tz": -480
    },
    "id": "bf2DjO6CP6TF",
    "outputId": "ec4ed75e-e813-4226-f85a-0baab3ff4354"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.Variable 'dense_1/kernel:0' shape=(6, 100) dtype=float32, numpy=\n",
      "array([[-0.18237606,  0.16415142,  0.20687856,  0.23396944,  0.09779547,\n",
      "        -0.14794639, -0.10231382, -0.22263053, -0.0950674 ,  0.18697281,\n",
      "         0.20488529, -0.04037735, -0.19727436,  0.0979359 , -0.1759503 ,\n",
      "         0.22504129,  0.21929003, -0.1273948 , -0.13652515,  0.02981101,\n",
      "         0.14656503,  0.20608391,  0.14076535, -0.02625689, -0.00161622,\n",
      "        -0.01449171,  0.23303385,  0.14593105,  0.11570902, -0.03970808,\n",
      "        -0.05525994, -0.20392904, -0.10306785,  0.21736331,  0.10087742,\n",
      "        -0.14146385,  0.03447478,  0.01457174, -0.06794603,  0.1030371 ,\n",
      "        -0.15175559,  0.22587933,  0.0804611 ,  0.21479838, -0.11029668,\n",
      "         0.22146653, -0.07499251,  0.1368954 , -0.13015983, -0.12019924,\n",
      "         0.21677957, -0.09586674, -0.05949883,  0.22539525,  0.2289827 ,\n",
      "        -0.02051648,  0.01296295,  0.16009761,  0.10034381,  0.12798755,\n",
      "        -0.10539538,  0.11883061,  0.07966466, -0.22101976,  0.12746729,\n",
      "        -0.1093536 , -0.16521278,  0.20071043,  0.16937451,  0.01447372,\n",
      "         0.16793476, -0.13962969,  0.1615852 , -0.10127702, -0.21089599,\n",
      "        -0.03635107, -0.2252161 , -0.02891247,  0.04012387,  0.1437303 ,\n",
      "        -0.14835042,  0.04761215,  0.00950299, -0.23300804,  0.09713729,\n",
      "         0.15262072, -0.00947247,  0.07256009, -0.15564013, -0.23770826,\n",
      "         0.20197298,  0.17501004,  0.16743289, -0.05297002,  0.06925295,\n",
      "         0.13787319, -0.00939476,  0.21161182, -0.14816652,  0.09728603],\n",
      "       [-0.23597604,  0.09226345, -0.21754897,  0.030596  , -0.02821516,\n",
      "        -0.11382222,  0.04664303,  0.03997506,  0.11674343,  0.17904802,\n",
      "         0.09352373, -0.06271012, -0.0995118 , -0.0839863 ,  0.19747855,\n",
      "         0.20034815, -0.00912318, -0.07400802,  0.1354406 , -0.10645141,\n",
      "         0.01427723, -0.23239797, -0.12351944,  0.20848687,  0.23319058,\n",
      "        -0.23722333, -0.05799152,  0.16170807,  0.03911252, -0.15837833,\n",
      "         0.2375672 , -0.0743922 ,  0.14655517,  0.08806615,  0.0797369 ,\n",
      "         0.1290553 , -0.08087416,  0.15930201, -0.00874732,  0.05047153,\n",
      "        -0.1929135 , -0.13928278,  0.17150857, -0.12328131,  0.18083079,\n",
      "        -0.00578046, -0.07798126,  0.0491005 , -0.18679146,  0.15352319,\n",
      "         0.08727174, -0.05599469,  0.23526688, -0.16455042, -0.1487674 ,\n",
      "        -0.14412951,  0.13929401, -0.09632155,  0.00167181,  0.1395758 ,\n",
      "        -0.04055098,  0.18666495, -0.02553804,  0.02356009,  0.08105092,\n",
      "         0.16715266,  0.08995526, -0.22296879,  0.06449543,  0.12188949,\n",
      "         0.23288779,  0.02098565, -0.07010749, -0.11215921,  0.23389243,\n",
      "         0.06119139, -0.0860803 , -0.12721145, -0.21600002, -0.20084816,\n",
      "        -0.16984203, -0.09152178, -0.1525074 , -0.09145156,  0.13034134,\n",
      "        -0.02183297,  0.04665013,  0.08182736, -0.18721978,  0.19582637,\n",
      "         0.02724861,  0.0457405 , -0.04569745,  0.04170756,  0.00739618,\n",
      "         0.06376933, -0.10515776,  0.15959997, -0.0694766 ,  0.02755009],\n",
      "       [-0.06319267,  0.13182057,  0.05506279, -0.20477852,  0.18970369,\n",
      "        -0.03943053,  0.11903851, -0.07155791,  0.04669012,  0.00812705,\n",
      "         0.05584623, -0.08371873, -0.15374976,  0.11677746, -0.12471052,\n",
      "        -0.16164276,  0.07642011,  0.20128562, -0.0017367 ,  0.1942391 ,\n",
      "        -0.00764962,  0.03465028, -0.18169758, -0.00213711,  0.05457829,\n",
      "        -0.07720377,  0.07186998, -0.16455126, -0.10122336,  0.17055278,\n",
      "         0.15423344,  0.23643543,  0.21533255, -0.08121212, -0.04754753,\n",
      "         0.08294679, -0.21654445, -0.01131117, -0.15386513, -0.06523256,\n",
      "        -0.19633041,  0.0248922 , -0.21637684,  0.00159171, -0.07713228,\n",
      "         0.22782163, -0.00349559, -0.16974016, -0.12448873,  0.17258836,\n",
      "        -0.01981039,  0.05522637,  0.12275855, -0.04751714,  0.05332409,\n",
      "        -0.07889435, -0.18033162,  0.06372721,  0.08295502, -0.19362703,\n",
      "        -0.0565086 ,  0.0755374 , -0.04072852,  0.07242356, -0.13613802,\n",
      "         0.20004983,  0.21880187,  0.14566241, -0.12664501,  0.21120678,\n",
      "         0.07900251,  0.00621621, -0.12596728,  0.11392109,  0.18359078,\n",
      "         0.07022355,  0.10555948, -0.22412848, -0.12310904,  0.1573142 ,\n",
      "        -0.11818868,  0.16200732,  0.15375106, -0.06835161, -0.2047078 ,\n",
      "         0.04541059,  0.1585447 ,  0.05231826,  0.22327141, -0.06841837,\n",
      "        -0.18981642,  0.08554949, -0.07791071,  0.12778889, -0.05718628,\n",
      "        -0.15301189,  0.17464243, -0.02707547,  0.23768233, -0.17579341],\n",
      "       [-0.09624396,  0.06217782, -0.02595268,  0.18477325, -0.10499094,\n",
      "         0.02485277, -0.13052183,  0.14919181,  0.13326867, -0.02644244,\n",
      "        -0.05067481,  0.218336  ,  0.07208259,  0.11023434,  0.09559919,\n",
      "         0.1890675 ,  0.16497074,  0.10391684, -0.10114242, -0.21803345,\n",
      "         0.20411105,  0.13692127, -0.15716076,  0.15078364,  0.04168098,\n",
      "        -0.21071486, -0.13353205, -0.16932839, -0.1560562 , -0.10680643,\n",
      "        -0.16614452,  0.10001676, -0.21421039, -0.06145182,  0.19260494,\n",
      "         0.17225821, -0.08490936, -0.05574624, -0.08104928, -0.0988984 ,\n",
      "         0.02178408, -0.16698101, -0.21433915,  0.0558214 , -0.00975212,\n",
      "         0.10072403, -0.02631079,  0.10000359,  0.15034248,  0.13407217,\n",
      "        -0.00151129, -0.0836809 ,  0.18076299,  0.15085675, -0.23618354,\n",
      "         0.05001326, -0.18851772, -0.11100499, -0.15566462, -0.20365193,\n",
      "        -0.2248187 , -0.07087977, -0.1306392 ,  0.0809411 , -0.00025243,\n",
      "         0.22963573,  0.01235358, -0.17157783, -0.2256927 , -0.17929551,\n",
      "         0.21892004, -0.23259266, -0.11126326, -0.08467883, -0.10183315,\n",
      "         0.2306784 , -0.09867099, -0.10394485, -0.10634935,  0.03752603,\n",
      "         0.17235063,  0.06114869,  0.022352  ,  0.15641572,  0.15094145,\n",
      "        -0.19431974,  0.15790848, -0.11217503,  0.19442542,  0.22548787,\n",
      "        -0.00768501,  0.158823  ,  0.14735304,  0.0210426 , -0.21737437,\n",
      "         0.08778961, -0.04887612,  0.0595323 ,  0.14631622,  0.02791832],\n",
      "       [-0.2108851 , -0.20682064, -0.20408219, -0.12765911, -0.06276481,\n",
      "        -0.13709348,  0.21450092,  0.01130743, -0.08537285,  0.14092736,\n",
      "         0.05730419, -0.02708216, -0.15316312,  0.00443964,  0.02637617,\n",
      "         0.10603566,  0.04577096, -0.11787698,  0.1437973 ,  0.23720677,\n",
      "        -0.1250069 , -0.03472549,  0.14871027, -0.05276214, -0.01313959,\n",
      "        -0.01017034,  0.09595035, -0.04737265,  0.10132308, -0.1576525 ,\n",
      "         0.09889011,  0.0852475 , -0.07178378, -0.191459  , -0.15800317,\n",
      "         0.21022727, -0.22400676, -0.2242892 ,  0.20423825,  0.2276028 ,\n",
      "         0.10706128, -0.15622596, -0.13000225, -0.03294952,  0.15762375,\n",
      "         0.09382527, -0.14523874,  0.08037744, -0.00737762,  0.16303493,\n",
      "         0.21322279, -0.20615923,  0.22627167, -0.01579981,  0.01377739,\n",
      "        -0.05311473,  0.0315261 , -0.05921595,  0.17797594,  0.21337356,\n",
      "        -0.00148429, -0.2224096 ,  0.17472763, -0.02621673,  0.12086569,\n",
      "         0.2221802 , -0.21260785,  0.19889398,  0.16766582,  0.11372416,\n",
      "         0.22781326, -0.17956433,  0.15436538,  0.20817696,  0.20706873,\n",
      "         0.17290573,  0.17291696,  0.0859191 , -0.17851931, -0.01899339,\n",
      "         0.06818415, -0.02687207, -0.09205528,  0.23415436,  0.09353746,\n",
      "        -0.09337692, -0.22046927,  0.20821108, -0.07228482,  0.04738481,\n",
      "        -0.13621926, -0.19762638, -0.13899282, -0.08098926,  0.130073  ,\n",
      "        -0.10348159, -0.07493602, -0.1722112 , -0.23290877,  0.18784209],\n",
      "       [ 0.13477843,  0.11936818, -0.21257897,  0.21244659, -0.18786472,\n",
      "        -0.06494723, -0.07063387, -0.07994832, -0.11256738, -0.22335076,\n",
      "        -0.02153319, -0.20943552, -0.21425952, -0.12278055, -0.00619341,\n",
      "        -0.09176037, -0.1766775 , -0.21622379, -0.04250833,  0.23764552,\n",
      "         0.21168886,  0.09459655, -0.07919639, -0.21559525,  0.20465617,\n",
      "        -0.20613717,  0.13103445,  0.21384992,  0.04693423,  0.20122723,\n",
      "         0.12190209,  0.22194327, -0.05410977, -0.1792583 , -0.03342254,\n",
      "         0.09272121,  0.06039228,  0.09666802, -0.22759588, -0.14688678,\n",
      "         0.12520896,  0.15474696, -0.23104139,  0.18017791, -0.02388267,\n",
      "        -0.01371126,  0.2352383 , -0.10501392,  0.01626216, -0.14222105,\n",
      "         0.13740788,  0.18499441,  0.03618436, -0.01862051, -0.1401035 ,\n",
      "        -0.01304157, -0.04905747, -0.07051091,  0.10759439, -0.08964662,\n",
      "        -0.01344521, -0.17841959, -0.17568308, -0.12892699,  0.11976974,\n",
      "         0.02280475,  0.16669382,  0.21027894,  0.21428709, -0.04820213,\n",
      "        -0.22136293, -0.13934767,  0.142024  , -0.07064074,  0.1470062 ,\n",
      "         0.00042979, -0.2371952 , -0.06649312,  0.10123204, -0.20473264,\n",
      "        -0.09161748,  0.20804678, -0.22195774, -0.09219673,  0.02311908,\n",
      "         0.13456099,  0.14470674, -0.05369592,  0.02126037,  0.0682667 ,\n",
      "         0.08384518,  0.17998771, -0.1927835 , -0.11473013, -0.01386146,\n",
      "        -0.10450925, -0.12111329, -0.2259491 ,  0.12304659, -0.04047236]],\n",
      "      dtype=float32)> <tf.Variable 'dense_1/bias:0' shape=(100,) dtype=float32, numpy=\n",
      "array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "      dtype=float32)>\n"
     ]
    }
   ],
   "source": [
    "print(layer.kernel, layer.bias)  # 也可以分别取出权重和偏置"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a8gZBodZQrmJ"
   },
   "source": [
    "## 二、实现自定义网络层\n",
    "实现自己的层的最佳方法是扩展tf.keras.Layer类并实现：\n",
    "\n",
    "- \\__init__()函数，你可以在其中执行所有与输入无关的初始化\n",
    "\n",
    "- build()函数，可以获得输入张量的形状，并可以进行其余的初始化\n",
    "\n",
    "- call()函数，构建网络结构，进行前向传播\n",
    "\n",
    "实际上，你不必等到调用build()来创建网络结构，您也可以在\\__init__()中创建它们。 但是，在build()中创建它们的优点是它可以根据图层将要操作的输入的形状启用后期的网络构建。 另一方面，在\\__init__中创建变量意味着需要明确指定创建变量所需的形状。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 437
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1064,
     "status": "ok",
     "timestamp": 1558273172643,
     "user": {
      "displayName": "Will Chen",
      "photoUrl": "",
      "userId": "01179718990779759737"
     },
     "user_tz": -480
    },
    "id": "SjQtJxARQLHY",
    "outputId": "237aa853-a405-418d-ad07-53705e1aa261"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[ 1.0200843  -0.42590106 -0.92992705  0.46160045  0.7518406   0.32543844\n",
      "   0.34020287  0.08215448  0.22044104 -0.5337319 ]\n",
      " [ 1.0200843  -0.42590106 -0.92992705  0.46160045  0.7518406   0.32543844\n",
      "   0.34020287  0.08215448  0.22044104 -0.5337319 ]\n",
      " [ 1.0200843  -0.42590106 -0.92992705  0.46160045  0.7518406   0.32543844\n",
      "   0.34020287  0.08215448  0.22044104 -0.5337319 ]\n",
      " [ 1.0200843  -0.42590106 -0.92992705  0.46160045  0.7518406   0.32543844\n",
      "   0.34020287  0.08215448  0.22044104 -0.5337319 ]\n",
      " [ 1.0200843  -0.42590106 -0.92992705  0.46160045  0.7518406   0.32543844\n",
      "   0.34020287  0.08215448  0.22044104 -0.5337319 ]\n",
      " [ 1.0200843  -0.42590106 -0.92992705  0.46160045  0.7518406   0.32543844\n",
      "   0.34020287  0.08215448  0.22044104 -0.5337319 ]], shape=(6, 10), dtype=float32)\n",
      "[<tf.Variable 'my_dense/kernel:0' shape=(5, 10) dtype=float32, numpy=\n",
      "array([[ 0.54810244,  0.042225  ,  0.25634396,  0.1677258 , -0.0361526 ,\n",
      "         0.32831818,  0.17709464,  0.46625894,  0.29662275, -0.32920587],\n",
      "       [ 0.30925363, -0.426274  , -0.49862564,  0.3068235 ,  0.29526353,\n",
      "         0.50076336,  0.17321467,  0.21151704, -0.26317668, -0.2006711 ],\n",
      "       [ 0.10354012, -0.3258371 , -0.12274069, -0.33250242,  0.46343058,\n",
      "        -0.45535576,  0.5332853 , -0.37351888, -0.00410944,  0.16418225],\n",
      "       [-0.4515978 ,  0.04706419, -0.42583126, -0.19347438,  0.54246336,\n",
      "         0.57910997,  0.01877069,  0.01255274, -0.14176458, -0.6309683 ],\n",
      "       [ 0.5107859 ,  0.23692083, -0.13907343,  0.51302797, -0.5131643 ,\n",
      "        -0.6273973 , -0.56216246, -0.23465535,  0.332869  ,  0.4629311 ]],\n",
      "      dtype=float32)>]\n"
     ]
    }
   ],
   "source": [
    "class MyDense(tf.keras.layers.Layer):\n",
    "    def __init__(self, n_outputs):\n",
    "        super(MyDense, self).__init__()\n",
    "        self.n_outputs = n_outputs\n",
    "    \n",
    "    def build(self, input_shape):\n",
    "        self.kernel = self.add_variable('kernel',\n",
    "                                       shape=[int(input_shape[-1]),\n",
    "                                             self.n_outputs])\n",
    "    def call(self, input):\n",
    "        return tf.matmul(input, self.kernel)\n",
    "layer = MyDense(10)\n",
    "print(layer(tf.ones([6, 5])))\n",
    "print(layer.trainable_variables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SOijVhx1EvdZ"
   },
   "source": [
    "## 三、网络层组合\n",
    "机器学习模型中有很多是通过叠加不同的结构层组合而成的，如resnet的每个残差块就是“卷积+批标准化+残差连接”的组合。\n",
    "\n",
    "在tensorflow2中要创建一个包含多个网络层的的结构，一般继承与tf.keras.Model类。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1081
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 855,
     "status": "ok",
     "timestamp": 1559143734067,
     "user": {
      "displayName": "Will Chen",
      "photoUrl": "",
      "userId": "01179718990779759737"
     },
     "user_tz": -480
    },
    "id": "xBDo-EYSUMkp",
    "outputId": "2cc12988-ea98-47b7-d6cf-2078930dcd8f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[[[0.79764616 1.0550306  0.9386751  1.1079601  0.9402881  0.99479383\n",
      "    0.9072118  0.5618475  0.9134829 ]\n",
      "   [0.79764616 1.0550306  0.9386751  1.1079601  0.9402881  0.99479383\n",
      "    0.9072118  0.5618475  0.9134829 ]\n",
      "   [0.79764616 1.0550306  0.9386751  1.1079601  0.9402881  0.99479383\n",
      "    0.9072118  0.5618475  0.9134829 ]\n",
      "   [0.79764616 1.0550306  0.9386751  1.1079601  0.9402881  0.99479383\n",
      "    0.9072118  0.5618475  0.9134829 ]\n",
      "   [0.79764616 1.0550306  0.9386751  1.1079601  0.9402881  0.99479383\n",
      "    0.9072118  0.5618475  0.9134829 ]\n",
      "   [0.79764616 1.0550306  0.9386751  1.1079601  0.9402881  0.99479383\n",
      "    0.9072118  0.5618475  0.9134829 ]\n",
      "   [0.79764616 1.0550306  0.9386751  1.1079601  0.9402881  0.99479383\n",
      "    0.9072118  0.5618475  0.9134829 ]\n",
      "   [0.79764616 1.0550306  0.9386751  1.1079601  0.9402881  0.99479383\n",
      "    0.9072118  0.5618475  0.9134829 ]\n",
      "   [0.83203167 0.9436392  1.0989372  1.2588525  0.8683256  1.1279813\n",
      "    0.7571581  0.47963202 0.88908756]]\n",
      "\n",
      "  [[0.79764616 1.0550306  0.9386751  1.1079601  0.9402881  0.99479383\n",
      "    0.9072118  0.5618475  0.9134829 ]\n",
      "   [0.79764616 1.0550306  0.9386751  1.1079601  0.9402881  0.99479383\n",
      "    0.9072118  0.5618475  0.9134829 ]\n",
      "   [0.79764616 1.0550306  0.9386751  1.1079601  0.9402881  0.99479383\n",
      "    0.9072118  0.5618475  0.9134829 ]\n",
      "   [0.79764616 1.0550306  0.9386751  1.1079601  0.9402881  0.99479383\n",
      "    0.9072118  0.5618475  0.9134829 ]\n",
      "   [0.79764616 1.0550306  0.9386751  1.1079601  0.9402881  0.99479383\n",
      "    0.9072118  0.5618475  0.9134829 ]\n",
      "   [0.79764616 1.0550306  0.9386751  1.1079601  0.9402881  0.99479383\n",
      "    0.9072118  0.5618475  0.9134829 ]\n",
      "   [0.79764616 1.0550306  0.9386751  1.1079601  0.9402881  0.99479383\n",
      "    0.9072118  0.5618475  0.9134829 ]\n",
      "   [0.79764616 1.0550306  0.9386751  1.1079601  0.9402881  0.99479383\n",
      "    0.9072118  0.5618475  0.9134829 ]\n",
      "   [0.83203167 0.9436392  1.0989372  1.2588525  0.8683256  1.1279813\n",
      "    0.7571581  0.47963202 0.88908756]]\n",
      "\n",
      "  [[1.0775117  1.1620466  0.72680396 1.0019443  1.2767658  1.1365149\n",
      "    1.1792164  1.0868194  1.0623009 ]\n",
      "   [1.0775117  1.1620466  0.72680396 1.0019443  1.2767658  1.1365149\n",
      "    1.1792164  1.0868194  1.0623009 ]\n",
      "   [1.0775117  1.1620466  0.72680396 1.0019443  1.2767658  1.1365149\n",
      "    1.1792164  1.0868194  1.0623009 ]\n",
      "   [1.0775117  1.1620466  0.72680396 1.0019443  1.2767658  1.1365149\n",
      "    1.1792164  1.0868194  1.0623009 ]\n",
      "   [1.0775117  1.1620466  0.72680396 1.0019443  1.2767658  1.1365149\n",
      "    1.1792164  1.0868194  1.0623009 ]\n",
      "   [1.0775117  1.1620466  0.72680396 1.0019443  1.2767658  1.1365149\n",
      "    1.1792164  1.0868194  1.0623009 ]\n",
      "   [1.0775117  1.1620466  0.7268039  1.0019443  1.2767658  1.1365149\n",
      "    1.1792164  1.0868194  1.0623009 ]\n",
      "   [1.0775117  1.1620466  0.7268039  1.0019443  1.2767658  1.1365149\n",
      "    1.1792164  1.0868194  1.0623009 ]\n",
      "   [0.87889266 0.9541194  0.8929231  0.96703756 1.0905087  1.0646607\n",
      "    0.9235744  0.9829142  1.1302696 ]]]], shape=(1, 3, 9, 9), dtype=float32)\n",
      "['resnet_block/conv2d_12/kernel:0', 'resnet_block/conv2d_12/bias:0', 'resnet_block/batch_normalization_v2_12/gamma:0', 'resnet_block/batch_normalization_v2_12/beta:0', 'resnet_block/conv2d_13/kernel:0', 'resnet_block/conv2d_13/bias:0', 'resnet_block/batch_normalization_v2_13/gamma:0', 'resnet_block/batch_normalization_v2_13/beta:0', 'resnet_block/conv2d_14/kernel:0', 'resnet_block/conv2d_14/bias:0', 'resnet_block/batch_normalization_v2_14/gamma:0', 'resnet_block/batch_normalization_v2_14/beta:0']\n"
     ]
    }
   ],
   "source": [
    "# 残差块\n",
    "class ResnetBlock(tf.keras.Model):\n",
    "    def __init__(self, kernel_size, filters):\n",
    "        super(ResnetBlock, self).__init__(name='resnet_block')\n",
    "        \n",
    "        # 每个子层卷积核数\n",
    "        filter1, filter2, filter3 = filters\n",
    "        \n",
    "        # 三个子层，每层1个卷积加一个批正则化\n",
    "        # 第一个子层， 1*1的卷积\n",
    "        self.conv1 = tf.keras.layers.Conv2D(filter1, (1,1))\n",
    "        self.bn1 = tf.keras.layers.BatchNormalization()\n",
    "        # 第二个子层， 使用特点的kernel_size\n",
    "        self.conv2 = tf.keras.layers.Conv2D(filter2, kernel_size, padding='same')\n",
    "        self.bn2 = tf.keras.layers.BatchNormalization()\n",
    "        # 第三个子层，1*1卷积\n",
    "        self.conv3 = tf.keras.layers.Conv2D(filter3, (1,1))\n",
    "        self.bn3 = tf.keras.layers.BatchNormalization()\n",
    "        \n",
    "    def call(self, inputs, training=False):\n",
    "        \n",
    "        # 堆叠每个子层\n",
    "        x = self.conv1(inputs)\n",
    "        x = self.bn1(x, training=training)\n",
    "        \n",
    "        x = self.conv2(x)\n",
    "        x = self.bn2(x, training=training)\n",
    "        \n",
    "        x = self.conv3(x)\n",
    "        x = self.bn3(x, training=training)\n",
    "        \n",
    "        # 残差连接\n",
    "        x += inputs\n",
    "        outputs = tf.nn.relu(x)\n",
    "        \n",
    "        return outputs\n",
    "\n",
    "resnetBlock = ResnetBlock(2, [6,4,9])\n",
    "# 数据测试\n",
    "print(resnetBlock(tf.ones([1,3,9,9])))\n",
    "# 查看网络中的变量名\n",
    "print([x.name for x in resnetBlock.trainable_variables])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jZqlHOMhOGXr"
   },
   "source": [
    "如果模型是线性的，可以直接用tf.keras.Sequential来构造。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 161
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 769,
     "status": "ok",
     "timestamp": 1559144265421,
     "user": {
      "displayName": "Will Chen",
      "photoUrl": "",
      "userId": "01179718990779759737"
     },
     "user_tz": -480
    },
    "id": "6vJ_tjNcMfof",
    "outputId": "d66e1c37-f911-475e-c08f-6500e90d791a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=1354, shape=(1, 2, 3, 3), dtype=float32, numpy=\n",
       "array([[[[-0.36850607, -0.60731524,  1.2792252 ],\n",
       "         [-0.36850607, -0.60731524,  1.2792252 ],\n",
       "         [-0.36850607, -0.60731524,  1.2792252 ]],\n",
       "\n",
       "        [[-0.36850607, -0.60731524,  1.2792252 ],\n",
       "         [-0.36850607, -0.60731524,  1.2792252 ],\n",
       "         [-0.36850607, -0.60731524,  1.2792252 ]]]], dtype=float32)>"
      ]
     },
     "execution_count": 9,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seq_model = tf.keras.Sequential(\n",
    "[\n",
    "    tf.keras.layers.Conv2D(1, 1, input_shape=(None, None, 3)),\n",
    "    tf.keras.layers.BatchNormalization(),\n",
    "    tf.keras.layers.Conv2D(2, 1, padding='same'),\n",
    "    tf.keras.layers.BatchNormalization(),\n",
    "    tf.keras.layers.Conv2D(3, 1),\n",
    "    tf.keras.layers.BatchNormalization(),\n",
    "    \n",
    "])\n",
    "seq_model(tf.ones([1,2,3,3]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "N26giRCjPXOR"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "002-custom_layers.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
